On alternative automated content evaluation measures

نویسنده

  • Rahul Katragadda
چکیده

In this draft we describe our TAC submissions and post-TAC experiments for Automated Evaluation of Summaries of Peers task of Text Analysis Conference (TAC). We approached the problem using two different approaches. Firstly, we use a generative modeling based approach to capture the sentence level presence of keywords in peer summaries and provide two fairly simple alternatives to identify keywords. Secondly, we used the Stanford dependency (SD) formalism to obtain a dependency recall based metric for summary evaluation. Our results show that the generative modeling approach is indeed promising and further investigation of keyword identification would obtain better results. For the Stanford-dependency based evaluation, performance has been similar to other dependency based evaluations of the likes of Basic Elements (BE) and DEPEval.

برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید

ثبت نام

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

منابع مشابه

Evaluation of Document Citations in Phase 2 Gale Distillation

The focus of information retrieval evaluations, such as NIST’s TREC evaluations (e.g. Voorhees 2003), is on evaluation of the information content of system responses. On the other hand, retrieval tasks usually involve two different dimensions: reporting relevant information and providing sources of information, including corroborating evidence and alternative documents. Under the DARPA Global A...

متن کامل

A Differential Word Use Measure for Content Analysis in Automated Essay Scoring

As part of its nonprofit mission, ETS conducts and disseminates the results of research to advance quality and equity in education and assessment for the benefit of ETS's constituents and the field. To obtain a PDF or a print copy of a report, please visit: Abstract This paper proposes an alternative content measure for essay scoring, based on the difference in the relative frequency of a word ...

متن کامل

On Automated Evaluation of Readability of Summaries: Capturing Grammaticality, Focus, Structure and Coherence

Readability of a summary is usually graded manually on five aspects of readability: grammaticality, coherence and structure, focus, referential clarity and non-redundancy. In the context of automated metrics for evaluation of summary quality, content evaluations have been presented through the last decade and continue to evolve, however a careful examination of readability aspects of summary qu...

متن کامل

Design and evaluation of validity of an electronic alternative and augmentative communication system for Persian-speaking children

Introduction: Due to the high prevalence of communication disorders, augmentative and alternative communication methods are one the options ahead to solve the problems of these people. Since there are no complex tools for Persian-speaking children with communication disorders, we decided to design communication assistant software for these children that produces sound output. Materials and Meth...

متن کامل

Automated quantification of sympathetic beat-by-beat activity, independent of signal quality.

Sympathetic nerve activity (SNA) can provide critical information on cardiovascular regulation; however, in a typical laboratory setting, adequate recordings require assiduous effort, and otherwise high-quality recordings may be clouded by frequent baseline shifts, noise spikes, and muscle twitches. Visually analyzing this type of signal can be a tedious and subjective evaluation, whereas objec...

متن کامل

ذخیره در منابع من


  با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید

برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید

ثبت نام

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

عنوان ژورنال:

دوره   شماره 

صفحات  -

تاریخ انتشار 2009